Docket No 

Inventor 

Title 



POU920020098US2 
R. C. Chu et al 
METHOD FOR COMBINED AIR AND 
LIQUID COOLING OF STACKED 
ELECTRONICS COMPONENTS 



APPLICATION FOR UNITED STATES 



LETTERS PATENT 



" Express Mail" Mailing Label No. 
Date of Deposit: March 30, 2004 



EL 965409139 US 



I hereby certify that this paper is being 
deposited with the United States Postal Service 
as "Express Mail Post Office to Addressee" service 
under 37 CFR 1.10 on the date indicated above 
and is addressed to: Mail Stop Patent Application, 
Commissioner for Patents, P.O. Box 1450, 
Alexandria, VA 22313-1450. 



Name: Sandra L. Kilmer 
Signature: 



INTERNATIONAL BUSINESS MACHINES CORPORATION 



METHOD FOR COMBINED AIR AND 
LIQUID COOLING OF STACKED ELECTRONICS COMPONENTS 

CROSS-REFERENCE TO RELATED APPLICATIONS 

[0001] This application is a divisional of US patent application serial no. 10/303,284, filed 
November 25, 2002 entitled "Method And Apparatus For Combined Air And Liquid Cooling Of 
Stacked Elecronics Components", the entirety of which is hereby incorporated herein by 
reference. 

FIELD OF THE INVENTION 

[0002] The present invention relates in general to systems and methods for cooling 
rack-mounted assemblages of individual electronics units, such as rack-mounted computer server 
units. 

BACKGROUND OF THE INVENTION 

[0003] The power dissipation of integrated circuit chips, and the modules containing the 
chips, continue to increase in order to achieve increases in processor performance. This trend 
poses a cooling challenge at both the module and system level. Increased air flow rates are 
needed to effectively cool high power modules and to limit the temperature of the air that is 
exhausted into the computer center. 

[0004] In many large server applications, processors along with their associated electronics 
(e.g. memory, disk drives, power, etc.) are packaged in removable drawer configurations stacked 
within a rack or frame. In other cases, the electronics may be in fixed locations within the rack 
or frame. Typically, the components are cooled by air moving in parallel air flow paths, usually 
front-to-back, impelled by one or more air moving devices (e.g., fans or blowers). In some cases 
it may be possible to handle increased power dissipation within a single drawer by providing 
greater air flow, through the use of a more powerful air moving device or by increasing the 
rotational speed (i.e. RPM) of an existing air moving device. However, this approach is 
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becoming unmanageable at the frame level in the context of the computer installation (i.e. data 
center). The sensible heat load carried by the air exiting the frame will eventually exceed the 
ability of the room air conditioning to effectively handle the load. This is especially true for 
large installations with "server farms" or large banks of computer frames close together. In such 
installations not only will the room air conditioning be challenged, but the situation may also 
result in recirculation problems with some fraction of the "hot" air exiting one frame being 
sucked into the air inlets of a nearby frame. Furthermore, while the acoustic noise level of a 
powerful (or higher RPM) air moving, device in a single drawer may be within acceptable 
acoustic limits, because of the number of air moving devices in the frame, the total acoustic noise 
at the frame level may not be acceptable. In addition, the openings required in the frame for the 
entry and exit of air flow makes it difficult, if not impossible, to provide effective acoustic 
treatment to reduce the acoustic noise level outside the frame. Finally, as operating frequencies 
continue to increase, we are approaching the point where electromagnetic (EMC) cross talk 
between tightly spaced computer frames is becoming a problem largely due to the presence of the 
openings in the covers required for air flow. 

[0005] Recently, there has been an attempt to address some of the defects noted above by 
combining the air cooling approach with an air-to-water heat exchanger fixed within the server 
cabinet below the frame electronics. An example of such a system is Sanmina's EcoBay™ 442. 
The present invention builds upon prior approaches in ways that will become apparent below. 

[0006] For the foregoing reasons, therefore, there is a need in the art for an improved and 
self-contained mechanism for cooling rack-mounted modular electronic units. Objects of the 
present invention include providing such a mechanism, while addressing, for example, the 
acoustic and EMC constraints noted above. 

SUMMARY 

[0007] The invention described here, which provides an air/liquid heat removal enclosure 
system with electromagnetic and acoustic noise control capability for electronic equipment, is an 
improvement over prior art systems. The invention utilizes an enclosure scheme within which a 
stack or stacks of electronic drawers are packaged. Air is circulated within the enclosure to cool 
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the electronics and is passed across one or more air-to-liquid finned-tube heat exchanger(s) 
(mounted to the side of the electronics in a first embodiment, and in front and/or back of the 
electronics in a second embodiment) to transfer the total system heat load to water which then 
exits the frame. Because the frame is closed and because the inner walls are lined with sound 
absorbent or deadening material, much of the acoustic noise caused by the air moving devices 
will be contained within the closed cabinet. Also, since the cabinet is closed with only minimal 
openings at the bottom of the frame for cables, it will also exhibit improved characteristics with 
respect to reducing both susceptibility to electromagnetic interference from outside the cabinet 
and electromagnetic radiation from within the cabinet. 

[0008] In one embodiment, the apparatus is described with a side-mounted air-to-liquid heat 
exchanger. In a second embodiment, the heat exchanger is in front of and/or behind the 
electronics and the air circulation is top to bottom, with a connecting duct in the bottom, a 
converging air flow supply plenum being formed in the front cover, a diverging air flow return 
plenum being formed in the back cover, with one or more optional air moving devices being 
located in the plenums; in a variant of the second embodiment, a connecting duct is in the top, 
with converging supply and diverging return air flow plenums; in yet another variant, connecting 
ducts are in both top and bottom and supply and return air flow plenums are doubly convergent 
and doubly divergent, respectively. Finally, a mechanism is described for automatically opening 
vent panels in the event of loss of coolant so that room air can be used for temporary cooling. 

[0009] The recitation herein of desirable objects which are met by various embodiments of the 
present invention is not meant to imply or suggest that any or all of these objects are present as 
essential features, either individually or collectively, in the most general embodiment of the 
present invention or in any of its more specific embodiments. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0010] The subject matter which is regarded as the invention is particularly pointed out and 
distinctly claimed in the concluding portion of the specification. The invention, however, both as 
to organization and method of practice, together with further objects and advantages thereof, may 
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best be understood by reference to the following description taken in connection with the 
accompanying drawings in which: 

[0011] Figure 1 is a conventional air-cooled frame with electronics in removable drawers; 

[001 lj Figure 2 is a cross sectional top view of a horizontal closed loop air flow heat removal 
system for electronics frame with side mounted cooling frame (water connections to the heat 
exchanger not shown); 

[0012] Figure 3 is an exploded view of Figure 2; 

[0013] Figures 4a, 4b, and 4c are more detailed views of the side mounting chilled air 
sub-frame;. 

[0014] Figure 5a is a cross-section side view of a cabinet and stack of electronics drawers in a 
second embodiment of the present invention; 

[0015] Figure 5b is a cross-section side view of a cabinet and stack of electronics drawers in a 
variant of the second embodiment of the present invention; 

[0016] Figure 6 is a top view of a cabinet with front and back covers open; 

[0017] Figure 7 is a rear cross-section view showing a monolithic air-to-liquid heat exchanger; 

[0018] Figure 8a is a front cross-section view showing a bank of individual air-to-liquid heat 
exchangers; 

[0019] Figure 8b is an isometric view where the bank of individual air-to-liquid heat 
exchangers are each individually hinged; 

[0020] Figure 9 is a cabinet configuration with a duct placed on top of the cabinet for air flow 
from a return duct to a supply duct; 
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[0021] Figure 10 is a cabinet configuration with ducts placed at both top and bottom of the 
cabinet for air flow from a return duct to a supply duct; 

[0022] Figure 1 1 a is a top cross-section view of a closed-loop air/liquid heat removal 
enclosure system with vent panels open for the second embodiment; 

[0023] Figure 1 lb is a side cross-section view of a closed-loop air/liquid heat removal 
enclosure system with vent panels open for the second embodiment; 

[0024] Figure 12a is a top view of a door with multiple automatically opening vent covers; 

[0025] Figure 12b is a view from the inside of a frame, looking outward, of a door with 
multiple automatically opening vent covers; 

[0026] Figure 13a is a detailed view of mechanisms for locking and opening door vent covers 
from inside a frame, looking outward; 

[0027] Figure 13b is a view from above the door mechanisms of Figure 13a; 

[0028] Figure 13c is an enlarged cross-section of the gasket seal between the vent covers and 
door surface; 

[0029] Figure 13d is a top cross-section view of the vent cover latch assembly; 

[0030] Figure 13e is an enlarged front view of the latch plunger engaged in the cover latch; 

[0031] Figure 14a is a cross-sectional top view of the first embodiment with an approach to 
handling the over-temperature condition tailored to the first embodiment; and 

[0032] Figure 14b is an enlargement of the retaining member. 
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DETAILED DESCRIPTION OF THE INVENTION 

[0033] As shown in Figure 1, in rack-mounted configurations typical in the prior art, a 
plurality of air moving devices (e.g. fan or blower 11) provides the forced air flow 15 needed to 
cool the electronic components 12 within the drawer 13. Cool air is taken in through the 
louvered covers 14 in the front of the frame and exhausted out of the louvered covers 16 in the 
back of the frame. 

[0034] A first embodiment of the present invention uses the principle of closed air circulation 
heat removal and introduces an air flow configuration coupled with the placement of the heat 
exchanger on the side of the electronics frame; it may be preferred in that it offers the following 
advantages: 

• The side-mounted heat exchanger is not readily susceptible to accidental damage due to 
water leakage when service is performed. 

• The height of the frame is not extended. This is important in a room with sprinkler heads 
less than nine feet from the floor as there is a safety specification requiring a minimum of 
1 8 inches from the top of the frame to the sprinkler head. 

• Side chilled air sub-frame is shipped separately and on its own casters greatly simplifying 
assembly in the field. 

• Condensation that may be produced by the air chilling heat exchanger is well away from 
the electronics. 

• The positioning of the heat exchanger allows for large dimensions with an associatedly 
large heat removal capability. 

[0036] Figure 2 shows a cross-sectional top view of the embodiment under discussion. In this 
embodiment, the path for the closed loop air flow 1 101 is horizontal in nature in that it passes 
from side to side. Specifically, air chilled by the heat exchanger 21 located in the side mounted 
chilled air sub-frame 1 102 is directed to the front of the electronics frame 10 within the front 
cover 3 1 . Heat dissipated by the electronics 12 is transferred to the air as it passes through the 
electronics frame. The air is redirected into the side mounted chilled air sub-frame 1 102 by the 
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back cover 32 thus completing the closed loop air flow. Air moving devices 25 may be added 
within the side-mounted chilled air sub-frame 1 102 if required to handle the additional pressure 
drop associated with the closed loop flow. Note that the heat exchanger 21 is positioned 
diagonally across the width of the sub-frame. This configuration assures the best heat transfer 
while minimizing air side pressure drop for a sub-frame of a given width. It is important to 
minimize the width of the sub-frame because it increases the total footprint of the electronics 
frame. 

[0037] To emphasize the nature of the side-mounted chilled air sub-frame, an exploded top 
view of the frame assembly is shown in Figure 3. Note that both the back 32 and front covers 31, 
as well as the chilled air sub-frame 1 102, may be shipped separately and assembled to the 
electronics frame in the field. 

[0038] And finally, Figures 4a, 4b, and 4c further illustrate the chilled air sub-frame 
illustrating that it is a separate frame with its own casters for easily moving into position along 
side the electronics frame. 

[0039] In accordance with a second embodiment of the present invention, a cross-sectional 
side view of the cabinet and stack of electronic drawers is shown in Figures 5a and 5b. As 
shown in Figure 5a, air-to-liquid heat exchangers 21 to remove heat from the closed loop air 
cooling stream are provided across the front (entrance) and/or back (exit) of the stack of 
electronics drawers. A closed air flow loop is created by the supply plenum 22, electronics 
drawers 13, return plenum 23, and the connecting duct 24 at the bottom of the cabinet. Although 
not shown, electrical cables may penetrate the connecting duct to provide signal I/O and 
electrical power from outside the cabinet. As shown in Figure 5b, auxiliary fans or blowers 25 
may be provided at the inlet and exit of the supply and return air plenums, respectively, to 
increase the total system air flow and overcome any additional pressure drops incurred due to 
flow through the plenums. It will be noted that a converging cross-section profile is utilized in 
the supply plenum 22 and a diverging cross-section profile is used in the return plenum 23. The 
converging and diverging profiles are formed by the addition of surrounding filler material 28, 
which may preferably be a sound absorbing material such as polyurethane SONEX classic 
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available from Silent Source Interior Acoustical Products of Northampton, MA. (This example 
is representative of a class of products serving similar functions and does not imply any 
particular requirement for the specific characteristic of this product.) This is done to make the 
flow through each drawer as nearly uniform as possible. It should also be noted that the plenums 
and heat exchangers in the left side and right side of the cabinet are part of and are incorporated 
within the covers (or doors) which may be opened to access the electronics drawers. It may be 
seen that casters are provided both beneath 29 the central portion of the frame containing the 
stack of electronics cabinets, and under 27 the front and rear doors 27 to support their weight as 
they are moved into the open position. In order to minimize or even eliminate electromagnetic 
interference with other electronic equipment in the computer space the cabinet's outer walls are 
to be made of a continuous sheet of metal (e.g sheet metal ) or a composite/laminate structure 
containing a continuous sheet of metal. The metal sheet is to cover nearly all if not all the 
surface area associated with the cabinet walls. Additionally, the sheet metal is to be electrically 
grounded. In addition, electrically conductive gasket or tape should be used when the enclosure 
cabinet is attached onto the electronics frame to reduce EMC leaks through the interface. This 
has been a standard process to install EMC-related covers for computing equipment. 

[0040] Figure 6 provides a top view of the cabinet with the front door 3 1 swung fully open and 
the rear door 32 partially open to demonstrate how maintenance would be performed on the 
computer system. An electronics drawer 13 is also shown partially open extending outward from 
the stack of electronics drawers. Note that under this condition the electronics in the unaffected 
drawers can continue to operate because nothing has been done to interrupt the air flow through 
said drawers. Temperatures may increase somewhat and the heat load to the room will also 
increase a bit, but since this condition will occur for a relatively short time period (30 minutes or 
less), there should be no perceivable impact to computer performance or reliability. 

[0041] Figure 7 shows a front cross-sectional view of the cabinet in a second embodiment. A 
liquid coolant (e.g., water) enters 41 and leaves 42 the cabinet through openings in the bottom. 
Figure 7 shows a single monolithic air-to-liquid heat exchanger 21 covering the entire stack of 
electronics drawers. Alternatively, Figure 8a shows a front cross-sectional view where a bank of 
individual air-to-liquid heat exchangers 21 are coupled to cover the stack of electronics drawers. 
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Figure 8b shows the hinging of the individual exchangers with one of them swung out so that a 
drawer 13 can be opened. This is done so that when a particular electronics drawer has to be 
pulled or removed from the electronics frame only the heat exchanger associated with the 
corresponding electronics drawer has to be repositioned. In both figures 7 and 8a, air moving 
devices 25 are shown at the bottom of the air flow duct spanning the width of the duct. 
Additionally, sound absorbing material 28 can be seen lining both side walls and the top and 
bottom of the cabinet. 

[0042] Figures 9 and 10 show variants of the second embodiments of the invention. Figure 9 
shows a cabinet configuration with the connecting path or duct 24 for air flow from the return 
duct to supply duct placed at the top of the cabinet. Figure 10 shows a cabinet with air flow 
connecting ducts 24 at both top and bottom of the cabinet. In this configuration essentially 
one-half of the total flow occurs through the top half of the cabinet and the other one-half 
through the bottom of the cabinet. Although the ducts will occupy more cabinet space, this 
configuration will provide the most uniform distribution of air flow across the stack of 
electronics drawers. It may be noted that in this latter configuration, the cross-section profile of 
the supply and return plenums are doubly convergent and doubly divergent respectively 71 . It 
may be readily appreciated that the cabinet can be constructed as integral to the frame comprising 
the stack of electronics components or can be constructed integrally with the surrounding cabinet. 
Alternatively, the surrounding cabinet can be constructed separately and later attached to the 
frame. 

[0043] It can be appreciated that while the doors are open, the electronics in unaffected 
drawers can continue to operate by taking in room air for cooling and exhausting the heated air to 
the room. However, no consideration has yet been given as to what action is necessary to 
provide continued cooling in the event of an interruption of the flow of cooling water to the heat 
exchangers in the closed frame. In this instance air would continue to circulate in the closed 
frame picking up heat from the electronics until the air temperature entering the electronics 
drawers becomes so high as to necessitate a shutdown of power. Of course, this course of action 
is highly unacceptable from the customer viewpoint as it directly impacts the conduct of business 
whatever it may be. 
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[0044] The refinement described below provides a mechanism for automatically opening vent 
panels in the covers to allow room air to pass through the electronic drawers in the event of a loss 
of water-cooling and still allow the computer to continue to operate without an excessive 
temperature build-up. Figures 1 la and 1 lb show an approach to this refinement tailored to the 
second embodiment of the present invention which approach is likewise usable in the first 
embodiment. Figure 14 shows an approach tailored to the first embodiment of the present 
invention. As can be readily appreciated, this approach is also usable in the second embodiment. 
Both approaches avoid swinging the entire door open, so reducing intrusion into the aisle space. 

[0045] Figure 1 1 a shows the top and Figure lib shows the side cross-section views of a 
closed-loop air liquid heat removal enclosure system with the vent panels in the doors open at 
each end of the computer frame. This is the configuration that would exist immediately 
following system detection of ah air circulation flow over-temperature due to the loss of 
water-cooling in terms of either unacceptably low water flow rates or excessive cooling water 
temperatures or a loss of the auxiliary fans or blowers. The vent panels 81 have automatically 
swung outward allowing room air to pass through cutouts in the left and right hand doors. Air is 
sucked through each electronics drawer by the fan or blower 1 1 residing within the drawer, 
thereby providing cooling to the electronic components within the drawer. The hot exhaust air is 
blown out into the room. 

[0046] Under normal operating conditions, the vent panels are closed flush against the doors 
allowing air to circulate within the system enclosure. Figure 12a shows a top view, and Figure 
12b shows a front view (looking from within the system enclosure) of a door with the vent 
panels. A soft rubber gasket seal 91 is provided around the perimeter of the cover panel on the 
side facing the door. When the panel is closed, the seal is of a sufficient quality to prevent 
ingress or egress of air through the cutouts in the door. Either a greater or lesser number of vent 
covers 81 and cutouts in the door than those shown may be employed. The front view shows the 
position of door latches 92 to keep the vent cover shut under normal operating conditions and the 
actuation cables 93 and solenoid 94 employed to open the latch mechanism(s) in the event of a 
circulating air flow over temperature condition. 
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[0047] The mechanisms for locking and opening the vent panels are shown in more detail in 
Figures 13a, 13b, 13c, 13d, and 13e. The top view 13b shows the door with vent covers 81 
closed as they would be in normal operation. The enlarged view of Figure 13c depicts the 
gasket seal 91 provided around the inside edge of each vent cover between the vent cover and 
door surfaces. When the vent covers are closed, a torsional spring (not shown) located co-axially 
with the vent cover hinge line is compressed. An outward force exerted onto the vent covers by 
the torsional spring will open the vent covers when a latch is released. The torsional spring will 
also keep the doors on the inlet air side from closing due to drafting. 

[0048] Figure 13d also shows a top cross-section view of the vent cover latch assembly. When 
the vent cover is closed, the latch plunger 103 is moved back against the spring 104 and then 
springs back into place in the cover latch 1 10 locking the vent cover closed. This arrangement 
makes it possible for a service technician to close each vent cover individually after correcting 
whatever condition caused the loss of water cooling incident. Figure 13e shows an enlarged 
front view of the latch plunger 103 engaged in the cover latch 1 10. A small diameter actuation 
cable 93 is attached to a rod projecting from the end of each plunger. The cable from each 
plunger goes over a plastic roller wheel 107 down to a common bar (111, Figure 13 a) near the 
bottom of the door. A single main cable 109 from the common bar connects to a solenoid 108 
located closer to the bottom of the door. In the event a circulating air flow over-temperature is 
detected, the solenoid is electrically actuated pulling down on the main cable and the common 
bar. Each individual cable moves downward by the same amount causing each latch plunger to 
withdraw from its vent cover latch. The vent covers are thereby allowed to spring open causing 
the system enclosure flow configuration to switch from the normal recirculating closed flow 
pattern to the emergency open flow-through mode. The cooling provided by the heat exchanger 
could fail in several ways. The chilled water supplied to the heat exchanger could fail. The heat 
exchanger supply line could plug due to excessive fouling. Finally, the hoses on the supply lines 
to the heat exchanger or the tubes within the heat exchanger could break resulting in loss of 
coolant. The action necessary to continue operation of the machine should any of these events 
occur is to open the covers such that room air could then supply the necessary air flow to cool the 
nodes. The loss of cooling provided by the heat exchanger would be sensed from a temperature 
sensor within each node and when multiple (two or more) of these sensors reach their limit, a 
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command would be sent to the door opening mechanism to open the door. Alternatively, a 
temperature sensor located upstream from the air inlet to the nodes could be used. In addition, 
leak sensors are installed in a pan installed beneath the heat exchanger, and if water is detected, 
the solenoid operated valves on the supply and return lines will be closed. 

[0049] Figures 14a and 14b illustrate the system described in the first embodiment above with 
the front and rear doors open. This is the configuration that would exist immediately following 
system detection of an air circulation flow over-temperature due to the loss of water-cooling in 
terms of either unacceptably low water flow rates or excessive cooling water temperatures or a 
loss of the auxiliary fans or blowers. As was described for Figure 13, two solenoid actuated 
latches would have been activated to allow the vent panels to swing open. In Figure 14a, the 
doors 31 and 32 have automatically swung outward allowing room air to pass through the 
rack(s). A retaining member 43 (chain, flexible rod, or the like) prevents the door from opening 
completely and blocking an aisle — as shown more particularly in Figure 14b. Air is sucked 
through each electronics drawer by the fan blower 1 1 residing within the drawer, thereby 
providing cooling to the electronic components within the drawer. The hot exhaust air is blown 
out into the room. Note that the front and rear doors 3 1 and 32 are preferably hinged on opposite 
sides as shown in Figure 14a. Because of this, hot air exhausted out of door 32 will not have a 
tendency to be sucked into door 31 and back into the system. 

[0050] Finally, it can readily be appreciated that the configuration described as the first 
embodiment is particularly well suited to a "field-upgrade" of an existing rack-mounted 
electronics system to an enclosed system with an air-to-liquid heat exchanger augmenting the 
pre-existing air-cooling system simply by constructing the sides, cover, and top remotely, then 
attaching them to the pre-existing unit, in place, and connecting a coolant supply. 

[0051] While the invention has been described in detail herein in accord with certain preferred 
embodiments thereof, many modifications and changes therein may be effected by those skilled 
in the art. Accordingly, it is intended by the appended claims to cover all such modifications and 
changes as fall within the true spirit and scope of the invention. 
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